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A METHOD AND SYSTEM FOR HANDWRITING RECOGNITION WITH 
5 SCROLLING INPUT HISTORY AND IN-PLACE EDITING 



TECHNICAL FIELD 

The present invention relates to a system and method by which 
computing devices may more easily recognize input from a user. In particular, 
10 the present invention relates to a system and method for implementing a 

handwriting recognition user interface for data entry with a scrolling recognition 
history and in-place editing. 

BACKGROUND ART 

As the components required to build a computer system have reduced in 

15 size, new categories of computer systems have emerged. One of the new 

categories of computer systems is the "palmtop" computer system. A palmtop 
computer system is a computer that is small enough to be held in the hand of a 
user. Most palmtop computer systems are used to implement various Personal 
Information Device (PID) applications such as an address book, a daily 

20 organizer, and electronic notepads. 

Personal Information Devices include the class of computers, personal 
digital assistants and electronic organizers that tend both to be physically 
smaller than conventional computers and to have more limited hardware and 
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data processing capabilities. PIDs include, for example, products sold by Palm, 
Inc. of Santa Clara, California, under such trademark as Pilot, and Pilot 1000, 
Pilot 5000, PalmPilot, PalmPilot Personal, PalmPilot Professional, Palm, and 
Palm III, Palm V, Palm VII, as well as other products sold under such trade 
names as WorkPad, Franklin Quest, and Franklin Convey. 

PIDs are generally discussed, for example, in U. S. Patent Nos. 
5,125,0398; 5,727,202; 5,832,489; 5,884,323; 5,889,888; 5,900,875; 6,000,000; 
6,006,274; and 6,034,686, which are incorporated herein by reference. PIDs 
typically include a screen and data processor, allowing the PID user to operate a 
substantial variety of applications relating to, for example: electronic mail, a 
calendar, appointments, contact data (such as address and telephone numbers), 
notebook records, expense reports, to do lists, or games. PIDs also often include 
substantial electronic memory for storing such applications as well as data 
entered by the user. Due to their substantial variety of applications and uses, 
personal information devices are becoming increasingly widely used. 

Since PIDs and other similar palmtop computer systems are very small, 
keyboards are generally not efficient input devices. For example, PIDs with 
keyboards have keyboards that are so small that a user cannot touch-type. 
Furthermore, to use a keyboard a user must either place the PID system down 
onto a flat surface so the user can type with both hands or hold the PID system 
with two hands and type with thumbs only. 

Instead of using a keyboard, many PIDs employ a stylus and a 
touchscreen or digitizer pad as an input system. The stylus and touchscreen 
combination works well for PIDs since the arrangement allows a user to hold the 
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PID system in one hand while writing with the stylus onto the touchscreen with 
the other hand. For example, to enter a large amount of text, the user employs 
the stylus with a touchscreen text entry area, using a variety of text input 
strokes. 

Prior art Figure 1 shows a diagram of the personal information device 100 
having a first touchscreen area 101 and a second touchscreen area 102. 
Personal information device 100 has two touchscreen areas, touchscreen area 

101 configured to recognize touches as the regular use of the stylus 103 as a 
pointing device from the user, and touchscreen area 102 configured to recognize 
touches as text input. Touchscreen areas 101-102 are designed to register and 
recognize strokes from the stylus 103. Typically, touchscreen area 101 is 
configured as an icon and menu touchscreen display area and the smaller 
touchscreen area 102 is configured as a text entry area. For example, strokes 
entered in the icon and menu area (touchscreen area 101) are interpreted as icon 
and/or menu manipulations and strokes entered in the text entry area 
(touchscreen area 102) are interpreted as text characters and/or punctuation 
(e.g., handwriting glyphs and the like). Touchscreen area 101 is generally for 
displaying large amounts of text (e.g., paragraph or more). Touchscreen area 

102 is generally for entering text using input strokes that are recognized as text 
characters (e.g., graffiti), using, for example, stylus 103 . 

There are a number of problems with this prior art approach. One 
problem is the fact that there exists no feedback in the text entry area with 
regard to the entered strokes and the resulting recognized character. As 
characters are entered by the user via stylus 103 and touchscreen area 102, 
there is no feedback within touchscreen area 102 with regard to the particular 
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input strokes of the user or the resulting recognized characters. Recognized 
characters resulting from input strokes in touchscreen area 102 are displayed in 
touchscreen area 101, with new characters being entered at an insertion point 
110 as they are recognized. This forces the user to divert attention from the 
touchscreen area 102 as input strokes are entered. For example, to check the 
accuracy of recognition, the user needs to focus attention on the upper 
touchscreen area 101. Consequently, the user is constantly looking back and 
forth between areas 101-102, checking the accuracy of the input strokes as they 
are being recognized. 

Another problem is the fact that there is no easy way to move the 
insertion point 110 to different locations within the recognized text. As 
currently implemented in prior art personal information device 100, the text 
entry point 110 is moved by touching the desired location within the recognized 
text with stylus 103. There is no easy way to move the insertion point around 
using the touchscreen area 101. If the insertion point needs to be moved, the 
stylus must be employed in upper touchscreen area 101, which requires hopping 
back and forth between areas 101-102 as input strokes are entered. 

Another problem is the fact there is no "in-place editing" of the recognized 
text. In prior art personal information device 100, there is no easy way to edit 
previously entered text in order to, for example, correct mistakes. To correct a 
mistake in previously entered text, the user needs to select the mistake using 
touchscreen area 101 by moving the insertion point to the location of the error 
(e.g., using the stylus 103), remove the mistake using the touchscreen area 102 
(e.g., backspacing), then enter the correct character using the touchscreen area 



PALM 3666 



4 



5/29/2001 



102, and finally look at the touchscreen area 101 to ensure the mistake was 
corrected. 

These problems limit the usability of the handwriting recognition system. 
Many users simply refuse to use the handwriting recognition. Many of those 
users that use it, do so only as a last resort. 

Thus what is required is a solution for providing active feedback with 
regard to input strokes and recognized characters as text is being entered by a 
user. What is required is a solution that does not force a user to divert attention 
between two different touchscreen areas as text is being entered. What is 
further required is a solution that provides for in-place editing of recognized 
text. The present invention provides a novel solution to the above requirements. 
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SUMMARY OF THE INVENTION 

The present invention provides a solution for providing active feedback 
with regard to input strokes and recognized characters as text is being entered 
by a user. The present invention provides a solution that does not force a user to 
5 divert attention between two different touchscreen areas as text is being 
entered. The present invention provides a solution that implements in-place 
editing of recognized text. 

In one embodiment, the present invention is implemented as a computer 

10 implemented touch screen hand writing recognition user interface method for a 
hand-held computer system. The touchscreen user interface method can be 
implemented on a PID (personal information device) and can be implemented on 
a palmtop computer system. A first touchscreen area is provided for accepting 
text input strokes. A second touchscreen area is provided for displaying 

15 recognized text from the text input strokes. The text input strokes are displayed 
in the first touchscreen area. The text input strokes are recognized and the 
resulting recognized text is displayed in the second touchscreen area. A portion 
of the recognized text is displayed in the first touchscreen area, wherein the 
portion of the recognized text is shown as the text input strokes are recognized. 

20 The portion of the recognized text displayed scrolls as the new text input strokes 
are recognized* The portion of the recognized text in the first touchscreen area 
can be displayed in a different format with respect to the recognized text in the 
second touchscreen area. The text input strokes in a first part of the first 
touchscreen area are graphically shown as they are being recognized by the 

25 computer system. 
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BRIEF DESCRIPTION OF THE DRAWTNttS 

The present invention is illustrated by way of example and not by way of 
limitation in the Figures of the accompanying drawings, in which like reference 
numerals refer to similar elements, and in which: 

Prior art Figure 1 shows a diagram of a prior art personal information 
device having a first touchscreen area and a second touchscreen area. 

Figure 2 shows an exemplary embodiment of a touchscreen equipped 
handheld PID configured to provide a touch screen hand writing recognition user 
interface in accordance with the present invention. 

Figure 3 shows a diagram of a PID in accordance with one embodiment of 
the present invention, wherein the size of the touchscreen display is maximized 
with respect to the overall size of the PID and stylus. 

Figure 4 is a function block diagram showing the components of an 
exemplary embodiment of the PID from Figure 3. 



Figure 5 shows a diagram of the user interface of the present invention as 
depicted on a touchscreen display of the PID from Figure 3. 
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Figure 6 shows diagrams depicting in-place editing in accordance with one 
embodiment of the present invention. 

Figure 7 shows diagrams depicting the coordination of the first 
touchscreen area, the scroll controller, and the insertion point in accordance 
with one embodiment of the present invention. 

Referring now to Figure 8, a flowchart of the steps of a handwriting 
recognition user interface process in accordance with one embodiment of the 
present invention is shown. 
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DETAILED DESCRIPTION OF THE INVENTION 

In the following detailed description of the present invention, numerous 
specific details are set forth in order to provide a thorough understanding of the 
present invention. However, it will be obvious to one skilled in the art that the 
present invention may be practiced without these specific details. In other 
instances well known methods, procedures, components, and circuits have not 
been described in detail as not to obscure aspects of the present invention 
unnecessarily. 

Some portions of the detailed descriptions which follow are presented in 
terms of procedures, logic blocks, processing, and other symbolic representations 
of operations on data bits within a computer memory. These descriptions and 
representations are the means used by those skilled in the data processing arts 
to convey most effectively the substance of their work to others skilled in the art. 
A procedure, logic block, process, step, etc., is here, and generally, conceived to 
be a self-consistent sequence of steps or instructions leading to a desired result. 
The steps are those requiring physical manipulations of physical quantities. 
Usually, though not necessarily, these quantities take the form of electrical or 
magnetic signals capable of being stored, transferred, combined, compared, and 
otherwise manipulated in a computer system. It has proven convenient at 
times, principally for reasons of common usage, to refer to these signals as bits, 
values, elements, symbols, characters, terms, numbers, or the like. 

It should be borne in mind, however, that all of these and similar terms 
are to be associated with the appropriate physical quantities and are merely 
convenient labels applied to these quantities. Unless specifically stated 
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otherwise as apparent from the following discussions, it is appreciated that 
throughout the present invention, discussions utilizing terms such as 
"implementing," "interpreting," "displaying," "configuring," "initializing," or the 
like, refer to the actions and processes of an embedded computer system, or 
similar embedded electronic computing device, that manipulates and transforms 
data represented as physical (electronic) quantities within the computer 
system's registers and memories into other data similarly represented as 
physical quantities within the computer system memories or registers or other 
such information storage, transmission or display devices. 

The present invention is a computer implemented touch screen hand 
writing recognition user interface method. The present invention provides a 
solution for providing active feedback with regard to input strokes and 
recognized characters as text is being entered by a user. The present invention 
provides a solution that does not force a user to divert attention between two 
different touchscreen areas as text is being entered. The present invention 
provides a solution that implements in-place editing of recognized text. 
Embodiments of the present invention and its benefits are further described 
below. 

It should be noted that the method and system of the present invention 
can be adapted to provide the user interface method for a number of different 
types of touchscreen equipped computing devices. Such devices include, for 
example, personal information devices (PIDs), palmtop computing devices, 
handheld cellular telephones (cellphones) and other types of mobile telephones, 
alphanumeric paging devices, and the like, which are equipped with, or coupled 
to, a touchscreen display. 
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Figure 2 shows an exemplary embodiment of a touchscreen equipped 
handheld PID 12 configured to provide a touch screen hand writing recognition 
user interface in accordance with the present invention. However, it should be 
5 noted that many touchscreen equipped electronic devices (e.g., digital cameras, 
pagers, palmtop computing devices, handheld cellular telephones (cellphones) 
and other types of mobile telephones, alphanumeric paging devices, and the like 
can be adapted to provide the hand writing recognition user interface 
functionality of the present invention. As depicted in Figure 1, PID 12 includes 
10 a touchscreen 16 for accepting input from a user, such as, for example, when the 
user touches or strokes touchscreen 16 with a stylus. 

Referring still to Figure 2, in one embodiment, the present invention is 
realized as a computer implemented method of generating a touch screen hand 

15 writing recognition user interface for a computer system. The touchscreen user 
interface functions by accepting user input on touchscreen area 16 and a touch 
screen area 17 of PID 12. The touchscreen areas 16-17 are configured to register 
events and interpret the events as user input. As used herein, "events" refer to 
touches and/or strokes on the surface of touchscreen areas 16-17 of sufficient 

20 magnitude in order to be detected by the hardware comprising touchscreen areas 
16-17. The hardware and software implementing the functionality of 
touchscreen areas 16-17 is configured for accepting text input strokes and for 
accepting icon manipulation strokes, such as, for example, where a user taps, 
strokes, etc., the surface of the touchscreen with a stylus. In implementing the 

25 user interface, PID 12 registers user input strokes on the touchscreen input area 
(i.e., touchscreen 16). 
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Figure 3 shows a diagram of a PID 5 in accordance with a second 
embodiment of the present invention, wherein the size of the touchscreen display 
7 is maximized with respect to the overall size of the PID 5 and stylus 8. In this 
embodiment, PID 5 utilizes a unified touchscreen display 7 having the first and 
5 second touchscreen areas 16-17. Touchscreen display 7 maximizes the amount 
of area of PID 5 dedicated to displaying a user interface by implementing areas 
16-17 within a single touchscreen, thereby avoiding the requirement for 
inclusion of separate dedicated touchscreens for areas 16-17, thus making 
maximum use of the limited touchscreen size available for PIDs, palmtop 
10 computers, cellphones, and the like. 

Figure 4 is a function block diagram showing the components of an 
exemplary embodiment of the PID 5. As depicted in Figure 3, PID 5 includes a 
processor, or controller, 28 that is capable of executing software for controlling 
15 the functionality of touchscreen 7. Touchscreen 7 and sensors 2-3 communicate 
with data interface circuitry 26 through a bus 34. The processor 28 is also 
connected through the bus 34 to user interface circuitry 36, a data storage 
module 38 and memory 40. As used herein, the data storage module 38 and 
memory 40 may both generally be referred to as part of the PID memory 41. 



20 



25 



The memory 40 may contain a specific remote control loading application 
42. The remote control loading application 42 may operate, for example, after 
the processor 28 receives a user input touch commanding the loading of a certain 
software application (e.g., address book). 

User interface circuitry 36 in the PID 5 includes hardware and software 
components that provide user input and output resources for functions in the 
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processor 28. The user interface circuitry 36 includes display output 50, display 
input 52, and additional input/output interface circuitry 54. 

The display output 50 preferably receives digital information representing 
5 graphical data from the processor 28 and converts the information to a graphical 
display, such as text and or/images, for display on touchscreen 7 (shown in 
Figure 3). The display input 52 may receive data inputs, such as graphical data 
inputs, from a user of the PID 5. As described above, the graphical data inputs 
are preferably entered by the user with a stylus on touchscreen 7, and may 
10 include text, drawings, or other objects that are capable of being graphically 
presented. 

Typically, the additional input/output interface 54 permits user input and 
commands to be input through buttons and similar devices on the PID, e.g., 
15 buttons for scrolling through data entries and activating applications. 

Alternatively, the input/output interface 54 may allow the PID 5 to accept audio 
data as well as other types of non-graphical data. For example, audio data 
signals (or picture telephone video input) may be entered through the additional 
input/output interface 54. 

20 

Figure 5 shows a diagram of the user interface of the present invention as 
depicted on a touchscreen display 7 of the PID 5. As depicted in Figure 5, the 
display 7 includes a first touchscreen area 501 and a second touchscreen area 
502. 

25 

In this embodiment, touchscreen area 501 is configured for accepting text 
input strokes. Touchscreen area 502 is configured for displaying recognized text 
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from the text input strokes. As depicted in Figure 5, touchscreen area 502 shows 
a first field 503 and a second field 504. Recognized text 506 is also shown. The 
"current" text input stroke 530 is displayed in a first part of touchscreen area 
501 to the right a scroll controller 520. Touchscreen area 501 shows a portion of 
the recognized text 531 in a second part of touchscreen area 501 to the left of a 
scroll controller 520. 

In entering text, the text input strokes are successively entered (e.g., 
input stroke 530) by a user and are successively recognized. The resulting 
recognized text (e.g., recognized text 531) is displayed in the second part of 
touchscreen area 501 to the left of the scroll controller 520. The portion of the 
recognized text 531 corresponds to the recognized text shown in touchscreen 
area 502. The recognized text 531 shown in touchscreen area 501 scrolls to the 
left as new text input strokes are entered and new characters are recognized. In 
this manner, the scroll controller 520 and the insertion point 510 both scroll as 
new characters are entered by the user. As shown in Figure 5, the recognized 
text 531 in touchscreen area 501 is rendered graphically in a larger font with 
larger spacing with respect to the recognized text 506 shown in touchscreen area 
502. This aids in-place editing, wherein the user corrects mistakes by "writing" 
directly over the incorrect character in touchscreen area 501. A small font is 
used in touchscreen area 502 in order to display a larger amount of recognized 
text. 

Draggable navigation of the recognized text in touchscreen area 502 is 
implemented, such that by dragging a boundary of the touchscreen area 501 
(e.g., left or right edge, top or bottom edge) the user can change the portion of the 
recognized text shown in the first touchscreen area 501. 
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Draggable scrolling of the recognized text in touchscreen area 502 is 
implementing, such that by dragging the scroll controller 520 left or right, the 
text shown in touchscreen area 501 scrolls correspondingly to the left or the 
5 right. This can be used to control which portion of the recognized text shown in 
touchscreen area 502 is shown in touchscreen area 501. 

To provide feedback to the user, the text input strokes in the touchscreen 
area 501 are graphically rendered (e.g., text input stroke 530) as they are being 

10 recognized by the computer system. For example, the text input stroke 530 is 
graphically rendered after having been entered by the user. The resulting 
recognized character is rendered directly above the scroll controller 520 and 
directly to the left of the insertion point 510. This provides visual feedback to 
the user, for example, allowing the user to correct subsequent input stroke 

15 technique to reduce recognition errors. 

Referring still to Figure 5, touchscreen area 501 includes a tab spot 541. 
Tab spot 541 functions by allowing the user to change the location of the text 
insertion point 510 with respect to, for example, the first field 503 and the 

20 second field 504. The user tapping the tab point 541 immediately jumps the 
insertion point 510 from, for example, the first field 503 to the second field 504. 
This aids the user in filling out electronic forms, etc. Additionally, touchscreen 
area 501 includes a caps spot 542. Caps spot 542 functions by allowing the user 
to change the case of the currently recognized character. For example, tapping 

25 on caps spot 542 allows the user to change the character at insertion point 510 
from a lowercase "a" to an upper case "A" conveniently. 
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Figure 6 shows diagrams depicting in-place editing in accordance with the 
present invention. The diagram of the left side of Figure 6 shows the display 7 
as an in-place edit is being made. The diagram on the right side of Figure 6 
shows the display 7 after the in-place edit has been made. Touchscreen area 501 
5 implements in-place editing by replacing one or more previously recognized 
characters shown on the left hand side of scroll controller 520, in this case 
character 602, with newly recognized one or more characters. This is done by 
the user "writing" directly over the previously recognized one or more characters. 
This is shown in the left diagram by the text input stroke 601 being made 

10 directly over the character 602, a "k" in this case. The text input stroke 601 is 
subsequently recognized, as an V in this case ? and replaces the previous 
character 602. This is shown in the right diagram. It should be noted that the 
insertion point 510 and the scroll controller 520 did not change their location. 
Writing over a recognized character replaces the character which the center of 

15 the text input stroke is closest to. This replacement does not alter the location of 
the insertion point 510. 

Figure 7 shows diagrams depicting the coordination of the first 
touchscreen area 501, the scroll controller 520, and the insertion point 510. The 

20 diagram on the left of Figure 7 shows insertion point 510 in the first field. The 
diagram on the right of Figure 7 shows insertion point 510 in the second field. 
Thus, it should be noted that the "focus" of touchscreen area 501 follows the 
insertion point 510. While the characters displayed in touchscreen area 501 can 
be changed by, for example, draggable navigation or draggable scrolling, the 

25 characters displayed in touch in area 501 can also be change simply by moving 
the insertion point 510 within touchscreen area 502 (e.g., touching with a 
stylus). 
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Referring now to Figure 8, a flowchart of the steps of a process 800 in 
accordance with one embodiment of the present invention is shown. Process 800 
shows the operating steps of the user interface method of the present invention 
5 as implemented on PID 5 of Figure 3. 

Process 800 begins in step 801 where text input strokes are accepted a 
first touchscreen area (e.g., touchscreen area 501) of PID 5. The first 
touchscreen area 501 is initialized to accept text input strokes from a user. In 
step 802, recognized text from the text input strokes are displayed in a second 
touchscreen area (e.g., touchscreen area 502). In step 803, the text input strokes 
are displayed as they are entered into touchscreen area 501 to provide feedback 
to the user with respect to character recognition. In step 804, text entry 
continues, with the recognizing of text input strokes and displaying of recognized 
text in touchscreen areas 501 and 502. As described above, touchscreen area 
501 displays a recognition history of the most recently entered characters, and 
thus comprises a smaller portion of a larger amount of text displayed in 
touchscreen area 502. The text displayed in touchscreen area 501 can be 
changed via, for example, draggable navigation or draggable scrolling. In step 
805, as text entry continues, the portion of the recognized text in touchscreen 
area 501 scrolls as new text input strokes are entered and recognized. 

Thus, the present invention is a computer implemented touch screen hand 
writing recognition user interface method. The present invention provides a 
25 solution for providing active feedback with regard to input strokes and 

recognized characters as text is being entered by a user. The present invention 
provides a solution that does not force a user to divert attention between two 
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different touchscreen areas as text is being entered. The present invention 
provides a solution that implements in-place editing of recognized text. 

The foregoing descriptions of specific embodiments of the present 
invention have been presented for purposes of illustration and description. They 
are not intended to be exhaustive or to limit the invention to the precise forms 
disclosed, and obviously many modifications and variations are possible in light 
of the above teaching. The embodiments were chosen and described in order 
best to explain the principles of the invention and its practical application, 
thereby to enable others skilled in the art best to utilize the invention and 
various embodiments with various modifications as are suited to the particular 
use contemplated. It is intended that the scope of the invention be defined by 
the claims appended hereto and their equivalents. 
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